Repository navigation
feat(common-utils): add a compact form of the pattern predicate - #3664
alex-fedotyev wants to merge 15 commits into
Conversation
templateToRegex and buildPatternPredicate build the whereSnippet that the MCP clickstack_event_patterns tool returns for each pattern. They live in the MCP server today. This adds them to @hyperdx/common-utils/dist/drain so the app can build the same filter. The app mines patterns with all ASCII punctuation as extra delimiters, so the separator class gets an extraDelimiters option. When every ASCII punctuation character is a delimiter, the class uses RE2's [:punct:] instead of listing 32 characters between every token. parsePatternPredicate reads a predicate back into its value expression and template. It only accepts predicates these functions built, so a hand-written match() is not mistaken for one. Without extraDelimiters the output is byte-identical to the MCP copy. The next change switches MCP over and deletes its copy. Part of FR-385. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
clickstack_event_patterns now imports templateToRegex, buildPatternPredicate and MAX_PREDICATE_TEMPLATE_LENGTH from @hyperdx/common-utils/dist/drain, and its own copy is deleted. MCP mines without extra delimiters, so the whereSnippet it returns is unchanged. The existing patternDrilldown tests import from common-utils and pass as they are. Part of FR-385. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
buildPatternPredicate repeats the separator class between every token. With all ASCII punctuation as delimiters that is about 30 characters per token. The app keeps pattern filters in the URL, and with this form two exclusions of long patterns already pass Node's default 16 KB header limit. buildCompactPatternPredicate builds the same match in a different shape: replaceRegexpAll() turns every separator run in the value into one space, and the regex matches tokens joined by single spaces, so the separator class appears once. On 8 templates mined from the demo logs, 8 exclusions take 11K characters in the URL instead of 104K, and each template matches the same number of events in ClickHouse with either form. The cost is speed: those 8 exclusions over 15 minutes of logs took 0.95 s instead of 0.45 s. parsePatternPredicate reads both forms. Part of FR-385. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
The latest updates on your projects. Learn more about Vercel for GitHub. 2 Skipped Deployments
|
🦋 Changeset detectedLatest commit: b6f789e The changes in this PR will be included in the next version bump. This PR includes changesets to release 4 packages
Not sure what this means? Click here to learn what changesets are. Click here if you're a maintainer who wants to add another changeset to this PR |
| return `match(${normalized}, '${escapeSqlString(compactRegex(template))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — The inverse of escapeSqlString now exists in three places
unescapeSql is new here, a copy of it is pasted into __tests__/pattern-predicate.test.ts:49, and filters.ts:190 (getBooleanOrUnquotedString) already does the same unescape inline. Add an exported unescapeSqlString next to escapeSqlString in core/utils.ts and use it in all three places, so the escape and unescape rules live in one file.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Hoisting an unescapeSqlString next to escapeSqlString and switching filters.ts over would touch files outside this PR, so that's left for a separate cleanup.
PR Review
1 finding(s): 🔴 0 critical · 🟠 0 major · 🔵 1 minor 1 posted as inline comment(s) on the changed lines. Severity is the reviewer's own estimate and is used for ordering, not filtering. No finding blocks merge automatically; a maintainer decides. |
Deep ReviewScope: ✅ No critical issues found. The equivalence logic is sound for all realistically-mined templates, SQL escaping is applied on every template-derived path, and the round-trip guard ( 🟡 P2 -- recommended
🔵 P3 nitpicks (3)
Reviewers (9): correctness, adversarial, security, testing, maintainability, kieran-typescript, api-contract, project-standards, previous-comments. Testing gaps:
No finding blocks merge automatically. A maintainer will expect P0/P1 findings in code this PR changes to be fixed; P2/P3 are your call -- fix or reply. Never fix findings about surrounding code here; reply instead. Do not widen the PR. How to respond |
- whitespace: 'python' matches templates mined by Drain3, whose str.split() also splits on \x1C-\x1F and NEL but not on BOM. - A delimiter longer than one character throws: Drain splits on it as a whole string, which a character class can't express. - Export ASCII_PUNCTUATION so the app builds its delimiter list from it. - Tests for the length limit, a round trip without options, and options that don't match in either direction. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mcp-shared-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…385-compact-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # packages/common-utils/src/__tests__/pattern-predicate.test.ts # packages/common-utils/src/drain/index.ts
- Run the compact-vs-full and round-trip tests under every option set (default, punctuation, a partial delimiter set, Python whitespace), with values that use non-ASCII separators. - Test a hand-written condition that only the round trip rejects. - Share the compact regex's start, end and wildcard between the builder and the parser. - Document when the two forms agree, and that valueExpression goes into the SQL as is. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Review findings outside the inline threads, fixed in e6804d4:
|
| return `match(${normalized}, '${escapeSqlString(compactRegex(template))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — The inverse of escapeSqlString is now written out by hand in three places
unescapeSql here, its copy in pattern-predicate.test.ts:31, and the inline .replace(/\\\\/g, '\\').replace(/''/g, "'") in filters.ts:190 all undo escapeSqlString (core/utils.ts:147). Fix: export one unescapeSqlString next to escapeSqlString in core/utils.ts and import it in all three places. The pattern for a quoted SQL literal, '((?:[^'\\]|\\\\|'')*)', now appears three times in this file (lines 150 twice, 189) and could become one shared constant.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
There was a problem hiding this comment.
Same answer as the earlier thread: hoisting the helper means touching core/utils.ts and filters.ts, outside this PR, and building the two parser regexes from a shared string would make them harder to read than the literals.
With whitespace: 'python' the template was still split on JS \s, which also splits on BOM. Drain3 keeps a BOM inside a token, so a template mined from a body that starts with one gave a regex its own events failed. Also test a round trip with quotes, backslashes and regex syntax in tokens. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mcp-shared-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…385-compact-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> # Conflicts: # packages/common-utils/src/drain/pattern-predicate.ts
- buildCompactPatternPredicate returns null for a template whose token holds one of the extra delimiters: normalizing would split the token, so the two forms would disagree. - Test templates with regex and SQL syntax in their tokens, so escaping in the compact form and its parser is covered. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
|
Second-round findings outside the inline threads:
That's the second round of automated findings, so per AGENTS.md I'm leaving any further ones for review. |
| return `match(${normalized}, '${escapeSqlString(compactRegex(template, options))}')`; | ||
| } | ||
|
|
||
| const unescapeSql = (literal: string) => |
There was a problem hiding this comment.
🔵 minor — unescapeSql is a third hand-written inverse of escapeSqlString, and the test file copies it again
Export an unescapeSqlString from packages/common-utils/src/core/utils.ts, next to escapeSqlString. Then use it here, in __tests__/pattern-predicate.test.ts:31, and in filters.ts:190 (getBooleanOrUnquotedString), which already does the same \\→\ and ''→' replacements inline.
Advisory. Fix if it is a small defect in code this PR changes; otherwise reply in the thread. Do not widen the PR or touch files it did not already change. How to respond
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…mcp-shared-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…385-compact-pattern-predicate Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
🟡 Tier 3 — StandardIntroduces new logic, modifies core functionality, or touches areas with non-trivial risk. Why this tier:
Review process: Full human review — logic, architecture, edge cases. Stats
|
Duplicate code analysisNo new duplicate code introduced by this PR. jscpd compared this branch against |
E2E Test Results✅ All tests passed • 466 passed • 1 skipped • 1003s
Tests ran across 4 shards in parallel. |
Adds
buildCompactPatternPredicate: the same match asbuildPatternPredicatein much shorter SQL, for pattern filters that live in the URL (#3665).parsePatternPredicatereads both forms. Part of FR-385.Stack
Part 3 of 4. All four target
main, and each is built on the one before it, so please merge them in order:Until #3663 merges, this diff also shows the PRs before it. This step on its own.
Summary
buildPatternPredicaterepeats the separator class between every token. With all ASCII punctuation as delimiters that is about 30 characters per token, before SQL escaping and URL encoding. The compact form turns every separator run in the value into one space, then matches tokens joined by single spaces, so the class appears once. ForGET api v1 users <*>:Why a second form
Measured on the OpenTelemetry demo logs, with 8 templates the app mined from 15 minutes of logs:
buildPatternPredicatebuildCompactPatternPredicateEach template also matched the same number of events with either form (15,198 to 30,056). Node's default header limit is 16 KB, so with the full form two exclusions of long patterns already break the page URL, while the compact form fits about ten. The price is about 2x on the exclusion filters. Keeping the compact form in the URL and querying with the full form would win that back; that is left for a follow-up.
Test plan
replaceRegexpAll, thenmatchparsePatternPredicateunder every option set; a hand-written condition is rejected both by the separator check and by the round tripyarn ci:lintandyarn ci:unitinpackages/common-utils(47 suites, 3,160 tests), knip,yarn dupes(no new duplicate code)